Method and system for error correction in automated wire contact insertion within a connector

ABSTRACT

A method, system and computer program product are provided for correction of automated insertion of a wire contact into a target hole of a connector. Methods include controlling a robot having an end-effector to: align the wire contact with the target hole of the connector; advance the wire contact toward and into the target hole of the connector; cease insertion in response to a force between the wire contact and the connector exceeding a predefined value; determining a depth of insertion; in response to the depth of insertion being above a predefined depth, perform a pull test on the inserted wire contact; in response to the depth of insertion being below a predetermined depth, identify an error condition using visual feedback; determine a number of corrective operations performed and perform error correction if the number is below a predefined number, while withdrawing the wire contact otherwise.

TECHNICAL FIELD

A method, system and computer program product are provided in accordancewith an example embodiment in order for automated alignment of wirecontacts with insertion holes of a connector and to provide errorcorrection thereof, and more particularly, to provide correction ofimproperly inserted wire contacts during automated contact insertioninto connectors.

BACKGROUND

Wire bundles consisting of a plurality of wires are utilized in avariety of industries to carry a myriad of different types of signals.The wire of a wire bundle assembly must frequently be terminated with awire contact and the resulting wire end is inserted into a wire contactinsertion hole of a connector, such as in a rubber grommet of aconnector. As each wire of a wire bundle is unique and may carry adifferent type of signal, the wire ends of a wire bundle assembly mustbe inserted into specific wire contact insertion holes of a connector inorder to make the proper connections.

The wire ends of a wire bundle assembly may be manually inserted intothe respective wire contact insertion holes defined by a connector. Aswire bundle assemblies commonly include dozens or possibly hundreds ofwires, this manual connection process may be relatively time consumingand error prone and, as a result, may increase the cost of the overallassembly including the wire bundle assembly. As such, automatedtechniques to insert the wire ends of a wire bundle assembly into thewire contact insertion holes of a connector have been developed in aneffort to reduce the time expended to make the connections and tocorrespondingly reduce the cost of the resulting assembly. However, wirebundle assembly machines generally require the connectors to be in avery restricted and controlled set of locations in order to increase thelikelihood that the wire ends of the wire bundle assembly may beproperly inserted into the wire contact insertion holes of theconnector. As such, wire bundle assembly machines limit the flexibilitywith which connectors may be presented and, as such, are not suitablefor all scenarios. Further, automated wire insertion techniques mayimproperly insert wire contacts into a connector, thereby halting theautomated process and requiring correction.

BRIEF SUMMARY

A method, system and computer program product are provided for automatedalignment of wire contacts with insertion holes of a connector and toprovide error correction thereof, and more particularly, to providecorrection of improperly inserted wire contacts during automated contactinsertion into connectors. Embodiments include a system for correctionof automated insertion of a wire contact into a target hole of aconnector, the system including: a robot having an end-effector, wherethe end-effector includes a wire gripper; a computing device, where thecomputing device is configured to: control the robot to align the wirecontact with the target hole of the connector; control the robot toadvance the wire contact toward the target hole of the connector and atleast partially insert the wire contact into the target hole; controlthe robot to cease insertion in response to a force between the wirecontact and the connector exceeding a predefined value; determine adepth of insertion of the wire contact within the target hole of theconnector; in response to the depth of insertion being above apredetermined depth, control the robot to perform a pull test of pullingthe wire contact away from the connector; in response to the depth ofinsertion being below a predetermined depth: identify an error conditionusing visual feedback; determine a number of corrective operationsperformed; perform an error correction operation in response to thenumber of corrective operations being below a predefined number; andmove the wire contact away from the target hole in response to thenumber of corrective operations being above the predefined number.

According to some embodiments, the computing device configured toperform an error correction operation includes the computing deviceconfigured to: control the robot to adjust an orientation of the wirecontact to be oriented perpendicular to a front surface of theconnector; and control the robot to re-insert the wire contact into thetarget hole of the connector. The computing device configured to adjustan orientation of the wire contact to be oriented perpendicular to afront surface of the connector, according to some embodiments, includesa computing device configured to: determine an orientation of the wirecontact relative to the connector; project the orientation of the wirecontact onto a plane perpendicular to an orientation of the wire gripperto identify a resulting vector; and control the robot to adjust theorientation of the wire contact according to the resulting vector to beoriented perpendicular to a front surface of the connector. According tosome embodiments, the computing device configured to adjust anorientation of the wire contact to be oriented perpendicular to a frontsurface of the connector includes a computing device configured to:determine an orientation of the wire contact relative to the connector;project the orientation of the wire contact onto a plane perpendicularto an orientation of the wire gripper to identify a resulting vector;increase the resulting vector by a predetermined amount; and control therobot to adjust the orientation of the wire contact according to theincreased resulting vector to be oriented perpendicular to a frontsurface of the connector.

The computing device configured to move the wire contact away from thetarget hole, according to some embodiments, includes the computingdevice configured to: control the robot to re-align the wire contactwith the target hole; control the robot to advance the wire contacttoward the target hole of the connector and at least partially insertthe wire contact into the target hole; control the robot to ceaseinsertion in response to a force between the wire contact and theconnector exceeding a predefined value; determine a depth of insertionof the wire contact within the target hole of the connector; in responseto the depth of insertion being above a predetermined depth, control therobot to perform a pull test of pulling the wire contact away from theconnector; and provide for indication of successful insertion of thewire contact into the target hole of the connector in response to thepull test satisfying predetermined criteria. The predetermined criteriaof some embodiments includes movement of the wire contact less than apredefined amount out of the target hole in response to a pull forceapplied to the wire contact of at least a predetermined force.

According to an example embodiment, the end effector includes one ormore image acquisition devices for capturing images of the wire contactand the connector from at least two different angles. The computingdevice of an example embodiment configured to control the robot to alignthe wire contact with the target hole of the connector includes thecomputing device configured to: acquire images from at least twodifferent angles using the one or more image acquisition devices;identify a location of the target hole; identify an alignment directionto align the wire contact with the target hole of the connector; andcontrol the robot to move the wire contact in the alignment direction ina plane orthogonal to an axis along which the wire contact extends. Thedepth of insertion of the wire contact is determined, according to someembodiments, based on an initial distance of the wire contact to asurface of the connector and a traveled distance of the robot duringinsertion. The initial distance of the wire contact to the surface ofthe connector is established using images of the wire contact and theconnector from at least two different angles.

Embodiments provided herein include a method for correction of automatedinsertion of a wire contact into a target hole of a connector, themethod including: controlling a robot having an end-effector to alignthe wire contact with the target hole of the connector using a wiregripper of the end-effector; controlling the robot to advance the wirecontact toward the target hole of the connector and at least partiallyinsert the wire contact into the target hole; control the robot to ceaseinsertion in response to a force between the wire contact and theconnector exceeding a predefined value; determining a depth of insertionof the wire contact within the target hole of the connector; in responseto the depth of insertion being above a predefined depth, controllingthe robot to perform a pull test of pulling the wire contact away fromthe connector; in response to the depth of insertion being below apredetermined depth: identify an error condition using visual feedback;determining a number of corrective operations performed; performing anerror correction operation in response to the number of correctiveoperations being below a predetermined number; and moving the wirecontact away from the target hole in response to the number ofcorrective operations being above the predefined number.

According to some embodiments, performing an error correction operationincludes: controlling the robot to adjust an orientation of the wirecontact to be oriented perpendicular to a front surface of theconnector; and controlling the robot to re-insert the wire contact intothe target hole of the connector. Adjusting an orientation of the wirecontact to be oriented perpendicular to a front surface of theconnector, in some embodiments, comprises: determining an orientation ofthe wire contact relative to the connector; projecting the orientationof the wire contact onto a plane perpendicular to an orientation of thewire gripper to identify a resulting vector; and controlling the robotto adjust the orientation of the wire contact according to the resultingvector to be oriented perpendicular to a front surface of the connector.According to an example embodiment, adjusting an orientation of the wirecontact to be oriented perpendicular to a front surface of the connectorincludes: determining an orientation of the wire contact relative to theconnector; projecting the orientation of the wire contact onto a planeperpendicular to an orientation of the wire gripper to identify aresulting vector; increasing the resulting vector by a predeterminedamount; and controlling the robot to adjust the orientation of the wirecontact according to the increased resulting vector to be orientedperpendicular to a front surface of the connector.

According to an example embodiment, moving the wire contact away fromthe target hole further includes: controlling the robot to re-align thewire contact with the target hole; controlling the robot to advance thewire contact toward the target hole of the connector and at leastpartially insert the wire contact into the target hole; controlling therobot to cease insertion in response to a force between the wire contactand the connector exceeding a predefined value; determining a depth ofinsertion of the wire contact within the target hole of the connector;in response to the depth of insertion being above a predetermined depth,controlling the robot to perform a pull test of pulling the wire contactaway from the connector; and providing for an indication of successfulinsertion of the wire contact into the target hole of the connector inresponse to the pull test satisfying predetermined criteria.

Embodiments provided herein include an apparatus having at least oneprocessor and at least one memory including computer program code, theat least one memory and computer program code configured to, with theprocessor, cause the apparatus to at least: control a robot having anend effector to align the wire contact with the target hole of theconnector using a wire gripper of the end effector; control the robot toadvance the wire contact toward the target hole of the connector and atleast partially insert the wire contact into the target hole; controlthe robot to cease insertion in response to a force between the wirecontact and the connector exceeding a predefined value; determine adepth of insertion of the wire contact within the target hole of theconnector; in response to the depth of insertion being above apredetermined depth, control the robot to perform a pull test of pullingthe wire contact away from the connector; in response to the depth ofinsertion being below a predetermined depth: identify an error conditionusing visual feedback; determine a number of corrective operationsperformed; perform an error correction operation in response to thenumber of corrective operations being below a predefined number; andmove the wire contact away from the target hole in response to thenumber of corrective operations being above the predefined number.

Causing the apparatus to perform an error correction operation includes,in some embodiments, causing the apparatus to: control the robot toadjust an orientation of the wire contact to be oriented perpendicularto a front surface of the connector; and control the robot to re-insertthe wire contact into the target hole of the connector. According tosome embodiments, causing the apparatus to adjust an orientation of thewire contact to be oriented perpendicular to a front surface of theconnector includes causing the apparatus to: determine an orientation ofthe wire contact relative to the connector; project the orientation ofthe wire contact onto a plane perpendicular to an orientation of thewire gripper to identify a resulting vector; and control the robot toadjust the orientation of the wire contact according to the resultingvector to be oriented perpendicular to a front surface of the connector.

According to some embodiments, causing the apparatus to control therobot to align the wire contact with the target hole of the connectorincludes causing the apparatus to: acquire images from at least twodifferent angles using one or more image acquisition devices; identify alocation of the target hole; identify an alignment direction to alignthe wire contact with the target hole of the connector; and control therobot to move the wire contact in the alignment direction in a planeorthogonal to an axis along which the wire contact extends. According toan example embodiment, causing the apparatus to move the wire contactaway from the target hole includes causing the apparatus to: control therobot to re-align the wire contact with the target hole; control therobot to advance the wire contact toward the target hole of theconnector and at least partially insert the wire contact into the targethole; control the robot to cease insertion in response to a forcebetween the wire contact and the connector exceeding a predeterminedvalue; determine a depth of insertion of the wire contact within thetarget hole of the connector; in response to the depth of insertionbeing above a predetermined depth, control the robot to perform a pulltest of pulling the wire contact away from the connector; and providefor indication of a successful insertion of the wire contact into thetarget hole of the connector in response to the pull test satisfyingpredetermined criteria.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described certain example embodiments of the presentdisclosure in general terms, reference will hereinafter be made to theaccompanying drawings which are not necessarily drawn to scale, andwherein:

FIG. 1 is a perspective view of a connector according to an exampleembodiment of the present disclosure;

FIG. 2 is a front view of the connector of FIG. 1 according to anexample embodiment of the present disclosure;

FIG. 3 is a block diagram of the system that may be specificallyconfigured in accordance with an example embodiment of the presentdisclosure;

FIG. 4 depicts a robot end-effector, wire gripper, and image acquisitiondevices according to an example embodiment of the present disclosure;

FIG. 5 illustrates images of a connector acquired by the imageacquisition devices of the robot end-effector of FIG. 4 according to anexample embodiment of the present disclosure;

FIG. 6 is a flowchart of a calibration routine for calibrating the imageacquisition devices relative to the wire gripper and robot end-effectoraccording to an example embodiment of the present disclosure;

FIG. 7 is a flowchart of a process for aligning a wire contact with atarget insertion hole according to an example embodiment of the presentdisclosure;

FIG. 8 illustrates the process to extract the wire contact direction andtip position from an image according to an example embodiment of thepresent disclosure;

FIG. 9 illustrates the process flow for detecting contact holes inconnectors according to an example embodiment of the present disclosure;

FIG. 10 is a flowchart of a process for aligning the wire contactdirection with a target hole of the connector according to an exampleembodiment of the present disclosure;

FIG. 11 illustrates a connector with a wire bundle attached theretousing example embodiments of the alignment technique described herein;

FIG. 12 is a flowchart of a process for aligning a wire contact with atarget hole of a connector according to an example embodiment of thepresent disclosure;

FIG. 13 is a process flow of a method for aligning and inserting wirecontacts with target holes of a connector according to an exampleembodiment of the present disclosure;

FIG. 14 is a flowchart of a process for insertion of a wire contactwithin a target hole of a connector according to an example embodimentof the present disclosure;

FIG. 15 is a flowchart of a process for automated error correctionaccording to an example embodiment of the present disclosure;

FIG. 16 illustrates a method of image processing to determine wirecontact orientation according to an example embodiment of the presentdisclosure;

FIG. 17 is a flowchart of a process for image processing to determinewire contact orientation according to an example embodiment of thepresent disclosure; and

FIG. 18 is a flowchart of a method for insertion of a wire contact intoa target hole of a connector with automated error correction accordingto an example embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure now will be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allaspects are shown. Indeed, the disclosure may be embodied in manydifferent forms and should not be construed as limited to the aspectsset forth herein. Rather, these aspects are provided so that thisdisclosure will satisfy applicable legal requirements. Like numbersrefer to like elements throughout.

A method, system, and computer program product are provided inaccordance with an example embodiment described herein for automatedalignment of wire contacts with insertion holes of a connector and toprovide error correction thereof, and more particularly, to providecorrection of improperly inserted wire contacts during automated contactinsertion into connectors. The process described herein detects wirecontact and insertion holes simultaneously usingrobotic-end-effector-mounted cameras. Using simultaneous detection,embodiments of the disclosed method provide feedback for correctivemovements of a robot arm used to insert the wire contacts into theinsertion holes of the connector. The movements of the robot arm alignthe wire contact with a target insertion hole for successful insertioninto an appropriate hole of a connector.

According to an example embodiment, during insertion of the wire contactinto the target connector insertion hole, a force feedback sensor of therobot arm or robotic end effector monitors the insertion force. If theforce exceeds a predetermined value, the insertion is temporarilystopped. A check of the insertion depth is performed based on an initialdistance to the connector and a travel distance of the robotic endeffector. The initial distance can be estimated through vision using theaforementioned cameras or based on measured robot movement relative tothe connector. If the insertion depth is above a minimal distance,established based on the type of wire contact and connector, a pull testis carried out. If the depth is below the predetermined value, the wireconnector is identified as stuck and an error-correction maneuver isneeded. The correction of the wire contact direction is performed onlyif the number of such corrections is below a threshold number ofattempts. If the number of such corrections exceeds the threshold numberof attempts, the robot end effector moves the wire contact outside theconnector hole and alignment of the wire contact with the connector holeis repeated.

The assembly of wire bundles including the attachment of one or morewire connectors to the wire bundle has traditionally been alabor-intensive process that is both time consuming and introducesopportunities for errors in the assembly. Embodiments described hereinenable the automatic assembly of wire bundles and their associated wireconnectors and provide a mechanism for automated error correction withrespect to insertion of wire contacts into connector holes. Inparticular, embodiments provide for the automatic insertion of wire endsinto connectors, and correcting errors of insertion automatically.Embodiments described herein may use a robot arm with a robot endeffector to insert the wires, supporting a flexible layout of connectorsand wires.

A method, system and computer program product are provided in accordancewith an example embodiment in order to identify wire contacts and wirecontact insertion holes defined by a connector in order to align andinsert the wire contacts into the wire contact insertion holes, and toprovide for error correction with respect to insertion of the wirecontacts into a respective target insertion hole of a connector.Although the method, system and computer program product may beconfigured to identify the wire contacts and wire contact insertionholes of a variety of different types of connectors, the connectorsgenerally define a plurality of wire contact insertion holes orconnector holes within a housing with the wire contact insertion holesbeing arranged in a predefined configuration. Different connectors mayinclude different numbers of wire contact insertion holes and mayinclude wire contact insertion holes arranged in differentconfigurations.

One example of a connector is depicted in FIGS. 1 and 2 in the form of aconnector 10. As shown, the connector 10 includes a housing 12 and arubber grommet 16 disposed therein. Although the housing 12 may beconfigured differently for other types of connectors, the housing of theconnector 10 of the embodiment of FIGS. 1 and 2 is externally threadedto facilitate, for example, the secure threaded engagement of a wirebundle assembly or another connector therewith. The connector 10 ofFIGS. 1 and 2 also includes a radially extending flange defining aplurality of openings 14, such as for receiving screws or otherfasteners for mounting the connector to an assembly. Although theconnector 10 of FIG. 1 has a cylindrical shape, the connector of otherexample embodiments may have different sizes and shapes. In regards tothe example connector of FIGS. 1 and 2, a rubber grommet 16 is disposedwithin the housing and the rubber grommet defines a plurality of wirecontact insertion holes 18. The wire contact insertion holes 18 definedby the rubber grommet 16 are configured, e.g., sized and shaped, suchthat a wire end consisting of a wire contact connected, e.g., crimped,to the end of a wire, is inserted into and mechanically retained withinthe wire contact insertion hole 18. In some, but not all embodiments,the rubber grommet may also include a plurality of wire contacts inalignment with respective wire contact insertion holes defined by therubber grommet such that the wire end may be brought into secureelectrical contact with a respective wire contact of the connector.

As shown by the example of the connector 10 of FIGS. 1 and 2, theplurality of wire contact insertion holes 18 defined by the rubbergrommet 16, are arranged in a predefined pattern. In some embodiments,not all of the wire contact insertion holes of a connector 10 will beutilized and, instead, only a subset of the wire contact insertion holeswill receive and make electrical connection with corresponding wire endsof the wire bundle assembly. As illustrated in FIG. 2, the wire contactinsertion holes 18 defined by the rubber grommet 16 that are not to beutilized in conjunction with a particular application may be eliminatedfrom further consideration by the insertion a plug 20 into therespective wire contact insertion hole defined by the rubber grommet.Although a connector 10 that may be analyzed in accordance with anexample embodiment of the present disclosure is depicted in FIGS. 1 and2 and will be described hereinafter, the method, system and computerprogram product of an example embodiment may be utilized in conjunctionwith a wide variety of other connectors and the connector is illustratedand described by way of example, but not of limitation.

The plugs 20 of a wire connector may be used to fill holes that may notbe used for the wire bundle being assembled. For example, a connectormay have twenty wire contact insertion holes 18; however, a wire bundlefeeding the connector 10 may include only 18 wires and correspondingwire contacts. In such an embodiment, the unused wire contact insertionholes may be plugged with plugs 20 such that there is less or possiblyno opportunity for water, moisture, or other corrosive/oxidizingsubstance to enter the connector and contaminate the wires and wirecontacts.

Referring now to FIG. 3, a system for identifying wire contact insertionholes 18 of a connector 10, inserting wire contacts into correspondingwire contact insertion holes 18, and providing error correction forautomated wire contact insertion is depicted. As shown, the system 30includes cameras 32 configured the acquire images of the connector 10.While plural cameras are indicated in FIG. 3, embodiments may employ asingle camera, or may employ a single camera operating with mirrors toprovide various perspectives of the connector 10 using a single camera.The cameras described herein are a type of image acquisition device,where a variety of image acquisition device types may be used in placeof a camera. Image acquisition devices, generally, acquire an image ofthe field of view of the device. A camera, as described herein, acquiresan image of the field of view in the visible light spectrum andprocesses the image accordingly. The cameras 32 may be configured toacquire a gray scale image of the connector 10. Alternatively, thecameras 32 may be configured to acquire color images of the connector10. In an embodiment in which color images of the connector 10 areacquired, the image associated with each different color channel of thecameras 32, such as the red, green and blue color channels, may beaveraged to create a composite image for subsequent analysis and review.Alternatively, the different color channels of the cameras 32 may beseparately analyzed. The cameras 32 are generally configured to acquireimages of the front face of the connector 10, such as shown in FIG. 2,such that the plurality of wire contact insertion holes 18 defined bythe rubber grommet 16 are clearly visible. The cameras 32 may also beconfigured to acquire images of the wire contacts during alignment ofthe wire contacts with the connector 10. As such, the image acquired bythe cameras 32 of an example embodiment may be acquired at a pluralityof angles to provide different perspectives of the connector 10 and wirecontacts.

In addition to the cameras 32, the system 30 of FIG. 3 includes acomputing device 34 configured to analyze the images of the connector 10acquired by the cameras and to identify wire contact insertion holes ofthe connector and wire contacts. The system 30 may also be configured toidentify plugs 20 within a connector 10. As also shown in FIG. 3, thesystem 30 of an example embodiment also includes or is in communicationwith a robot 44 and, more particularly, a robotic end effector that isutilized to insert wire ends/contacts into respective candidate contactinsertion holes of the connector 10 based upon the identification of thewire contact insertion holes of the connector and the wire contacts bythe computing device 34.

The computing device 34 may be configured in various manners and, assuch, may be embodied as a personal computer, a tablet computer, acomputer workstation, a mobile computing device such as a smartphone, aserver or the like. Regardless of the manner in which the computingdevice 34 is embodied, the computing device of an example embodimentincludes or is otherwise associated with processing circuitry 36, memory38, and optionally a user interface 40 and a communication interface 42for performing the various functions herein described. The processingcircuitry 36 may, for example, be embodied as various means includingone or more microprocessors, one or more coprocessors, one or moremulti-core processors, one or more controllers, one or more computers,various other processing elements including integrated circuits such as,for example, an ASIC (application specific integrated circuit) or FPGA(field programmable gate array), or some combination thereof. In someexample embodiments, the processing circuitry 36 is configured toexecute instructions stored in the memory 38 or otherwise accessible tothe processing circuitry. These instructions, when executed by theprocessing circuitry 36, may cause the computing device 34 and, in turn,the system 30 to perform one or more of the functionalities describedherein. As such, the computing device 34 may comprise an entity capableof performing operations according to an example embodiment of thepresent disclosure while configured accordingly. Thus, for example, whenthe processing circuitry 36 is embodied as an ASIC, FPGA or the like,the processing circuitry and, correspondingly, the computing device 34may comprise specifically configured hardware for conducting one or moreoperations described herein. Alternatively, as another example, when theprocessing circuitry 36 is embodied as an executor of instructions, suchas may be stored in the memory 38 the instructions may specificallyconfigure the processing circuitry and, in turn, the computing device 34to perform one or more algorithms and operations described herein.

The memory 38 may include, for example, volatile and/or non-volatilememory. The memory 38 may comprise, for example, a hard disk, randomaccess memory, cache memory, flash memory, an optical disc (e.g., acompact disc read only memory (CD-ROM), digital versatile disc read onlymemory (DVD-ROM), or the like), circuitry configured to storeinformation, or some combination thereof. In this regard, the memory 38may comprise any non-transitory computer readable storage medium. Thememory 38 may be configured to store information, data, applications,instructions, or the like for enabling the computing device 34 to carryout various functions in accordance with example embodiments of thepresent disclosure. For example, the memory 38 may be configured tostore program instructions for execution by the processing circuitry 36.

The user interface 40 may be in communication with the processingcircuitry 36 and the memory 38 to receive user input and/or to providean audible, visual, mechanical, or other output to a user. As such, theuser interface 40 may include, for example, a display for providing animage acquired by the camera 32 and/or an image visually depicting theclosest match between the candidate contacts and a predeterminedtemplate as described below. Other examples of the user interface 40include a keyboard, a mouse, a joystick, a microphone and/or otherinput/output mechanisms.

The communication interface 42 may be in communication with theprocessing circuitry 36 and the memory 38 and may be configured toreceive and/or transmit data, such as by receiving images from thecamera 32 and transmitting information, such as a list of candidatecontact insertion holes, contact ID numbers and locations of thecandidate contact insertion holes in a connector-based coordinatesystem, to a robot 44 and/or a robotic end-effector. Although referencedherein as candidate contact insertion holes, contact ID numbers andlocations of the candidate contact insertion holes, the list ofcandidate contact insertion holes, contact ID numbers and locations ofthe candidate contact insertion holes is to be interpreted so as to beassociated with the candidate contact insertion holes themselves and/orwire contacts aligned with the respective candidate contact insertionholes in those embodiments that include such wire contacts. Thecommunication interface 42 may include, for example, one or moreantennas and supporting hardware and/or software for enablingcommunications with a wireless communication network. Additionally oralternatively, the communication interface 42 may include the circuitryfor interacting with the antenna(s) to cause transmission of signals viathe antenna(s) or to handle receipt of signals received via theantenna(s). In some environments, the communication interface 42 mayalternatively or also support wired communication.

Referring now to FIG. 4, an example embodiment of a system performingthe methods described herein is shown including a robot end-effector100, which may include a tool head having three or more degrees offreedom, and image acquisition devices including a first camera 102 anda second camera 104. The robotic end-effector 100 may carry a wire 111in a wire gripper 108 including a wire contact 114 at the leading end ofthe wire 111. A connector 110 is disposed in a fixed location as it isapproached by the robotic end-effector 100. The connector is illustratedwith a single target insertion hole 116; however, the single targetinsertion hole is shown for ease of understanding as connectors willinclude a plurality of target insertion holes. The two cameras 102, 104are mounted on the robotic end-effector 100 in such a way as to viewboth the wire 111 including the wire contact and the connector 110simultaneously.

While the embodiment of FIG. 4 includes two cameras, embodiments mayinclude more cameras. Further, a single camera may be used inconjunction with mirrors to observe different perspectives of the wirecontact and the connector using the single camera. Capturing multipleperspectives, such as using two or more cameras, may enable accuratepositioning of the wire contact and the connector as they are joined.

According to example embodiments described herein, images are acquiredof the wire 111 and wire contact 114 along with the connector 110 frommore than one perspective. Using the different perspectives, a line isidentified that extends in the direction of the wire and wire contactand a hole in the connector that is the target hole for the wire isidentified. FIG. 5 illustrates images 120, 122 acquired by two differentimage acquisition devices, such as the cameras 102 and 104 of FIG. 4 ofthe wire 111 including wire contact 114 and the connector 110,specifically the identified target insertion hole 116 of the connectorinto which the wire 111 is to be inserted. A line, identified throughmultiple perspectives, provides at least a stereoscopic indication ofthe relationship between the wire contact and the target hole of theconnector into which the wire is to be inserted, and may be identifiedbased on the axial projection of the wire 111 and wire contact 114.Based on the identified line from the images, a movement command may becomputed that would place the hole on the line in at least two images.This may initially establish a rotation of the end-effector to bring thetip of the wire gripper 108 perpendicular to the connector surface. Toplace the hole on the line, a movement is established in parallel to theconnector surface to align the line with the appropriate target hole ofthe connector. A movement command is the desired displacement of therobot end-effector in three-dimensional cartesian space. Aligning thewire contact with the hole places the wire in a proper position toenable the robot to move the wire along the line toward the appropriatehole of the connector for insertion.

Embodiments described herein may calibrate the cameras ahead of usingthem to align the wire with the target hole of the connector. Thepurpose of the calibration is to compute a mapping of three dimensionalCartesian coordinates onto a two dimensional image coordinates. Thecalibration may be carried out before wires are fed to the robotic wiregripper of the end-effector. Calibration is not necessary before everywire insertion or before every connector change, but may be necessarywhen camera settings change, such as the focus, zoom, orientation, etc.

FIG. 6 depicts a process flow of an example calibration procedure. Theillustrated procedure uses a small calibration rod, which may be, forexample, a small plastic rod of around an inch in length and having adistinct tip such as a red tip. According to some embodiments, thecalibration rod may include a small sphere on a needle such as acomputer-aided measurement machine calibration stylus or simply a smalldot on a piece of paper. The calibration rod is used to provide aneasily identifiable point visible in each camera field of view from thedifferent camera perspectives. The calibration rod may be mounted firmlywith the tip facing up and within reach of the robot end-effector. Inpreparation for calibration, the robotic end-effector is advanced to bein front of the calibration rod, such that the calibration rod's tipposition mimics the expected position of a connector surface (e.g., thesurface into which the connector holes are formed). The calibrationprocedure may begin with a list of end-effector positions, as shown at130. One example for such a list are the three-dimensional coordinatesof nodes in a 3×3×3 cubic grid, where neighboring nodes are onecentimeter apart. A constraint for generating this list may be that foreach coordinate of the end-effector, the tip of the calibration rod mustbe visible in all camera images.

A new position of the end-effector positions is obtained at 132. Therobot loops through the list of end effector locations by moving therobot end-effector to the obtained position at 134, capturing images ofthe calibration rod at 136, finding the coordinates of the tip of thecalibration rod in both camera images at 138, and recording the endeffector position at 140. The process loops back to get a new positionfrom the list until all end-effector positions have been used forcalibration, or at least a predefined number of end-effector positionsto provide a satisfactory calibration. In each image acquired, thelocation of the tip of the calibration rod is identified. To identifythe tip, the image may be color filtered (e.g., by computing R−(G+B)/2for each pixel, where R, G, and B are the Red, Green, and Blue colorchannels, respectively). The average location of all pixels having anintensity value above a predefined value may then be computed. Thepredefined value may be chosen such that only the tip of the calibrationrod is selected. It may be beneficial to have a light source above thecalibration rod such that the tip is sufficiently illuminated and maystand out in the acquired images. The result of this calibrationprocedure are the two-dimensional image coordinates of the tip of thecalibration rod in each camera image.

Once the calibration routine of FIG. 6 is complete, the result is a listof three-dimensional end-effector positions in the end-effectorcoordinate frame and corresponding two-dimensional image coordinates foreach camera. This set of corresponding coordinates is used by analgorithm to calibrate each camera. For example, a Perspective-n-Point(PnP) algorithm may be used to calibrate each camera. A non-limitingexample of a PnP algorithm may be the UPnP+Gauss Newton algorithm. Theresult of this algorithm are two matrices for each camera: one thatencodes the intrinsic parameters (like the focal length) and one thatencodes the extrinsic parameters (position and pose of the camera).These matrices can be used to map a three-dimensional position in therobot's end-effector frame onto two-dimensional image coordinates. Usingthis calibration procedure, the camera locations do not need to be knownin advance.

Once the cameras have been calibrated, a wire contact held by therobotic end-effector may be aligned with a connector. FIG. 7 illustratesthe process of aligning a wire contact with a target insertion hole of aconnector. After a wire is grasped by the wire gripper of theend-effector, whether the wire is placed in the wire gripper orpicked-up by the wire gripper, images may be acquired by the camerasmounted on the end-effector at 150. In these images, the wire contact isdetected and its direction obtained as shown in 152. This operation isfurther described below. The robot may then move the wire contact to benear the connector surface at 154. In this position, the cameras againacquire images at 156 to include the wire contact and the connector.From these images, two processes are computed: first the direction ofthe wire contact is updated at 158; and second, connector holes aredetected at 160. By combining the output of these processes, the systemcomputes a movement command in the robot end-effector coordinates toalign the contact with a target hole at 162.

After the robot executes the first alignment step, camera images areagain acquired and both contact and target hole positions updated. Ifthis update yields a corrective movement command below a threshold(e.g., below 0.1 millimeters), the robot may not execute the correctionand instead proceeds to move the contact toward the connector surface.The direction of the movement of the wire contact toward the connectorsurface matches the contact's direction in three-dimensions as obtainedthrough the camera images. If the updated wire contact position yields acorrection above the threshold, the robot may then make the correctivemove and acquire new images, whereby the aforementioned process isrepeated until the correction is below the threshold.

The number of repetitions of the process of FIG. 7 may be limited, suchas to three attempts. After this limit, the robot may abort thealignment process and indicate an error, such as through an errormessage of a user interface. Alternatively, the robot may start againmoving the contact near the connector surface as before. Threesignificant elements of the alignment process of FIG. 7 are described ingreater detail below.

The detection of the wire contact is necessary to align the contact witha target hole and to understand the movement direction for the robotend-effector once the contact is aligned. FIG. 8 depicts the process toextract the direction of a wire contact from an image. In this exampleembodiment, the computing device 34, such as the processing circuitry36, may be configured to perform the various operations of extractingthe direction of a wire contact and tip position from the acquiredimages. The first operation is to extract a window of the image 166 inwhich the contact is expected to be. The image in the window may becolor filtered (e.g., by using a single color channel) to produce animage of only the color of interest. According to an example embodimentin which the wire contacts are gold in color, the image may be colorfiltered to find the gold colored areas at 167. A fit line isestablished at 168 based on the gold colored areas extending along alinear direction. The fit line constrains the area processed for edgedetection at 170, e.g., by using a 30-pixel wide corridor around theline. This corridor cuts out distracting edges in the background, e.g.,from other wires. For edge detection, the Canny edge detection algorithmmay be used. Non-limiting parameters of the Canny edge detector may be asigma or two for Gaussian blurring and thresholds of 0.005 and 0.015 foredge tracing. Second, to detect lines, a Hough transform may be carriedout on the edges as shown at 172. Third, using the resulting array fromthe Hough transform, the maximum may be found at 174 which correspondsto the longest line. By finding the maximum, the angle and orientationof the line and its distance from one of the image corners isidentified. Around the maximum, nearby maxima are sought with the sameline orientation. An example for these maxima is to have a value largerthan 0.5 times the maximum from the Hough transform. These maxima maycorrespond to parallel lines in the direction of the contact. The centerof the two extremal lines may be estimated as the position andorientation of the contact as shown at 176.

Once the direction of the contact is obtained, such as by the processingcircuitry 36 of the computing device 34, the location of the tip of thecontact is computed. To find the tip, the ends of all edge linesparallel to the contact may be determined. All ends may be projectedonto the contact line. The projection that is furthest away from theimage corner opposite the contact tip may be identified as the locationof the tip. This process to obtain the contact direction and tiplocation may be repeated for at least two camera images acquired fromdifferent perspectives.

In the same way as for the wire contact, though without using the goldcolor filter described above, the tip of the wire gripper can beobtained by the processing circuitry 36 of the computing device 34.Here, images may be analyzed without the wire contact inserted in thewire gripper. Such images may be acquired during calibration. Since thewire gripper is fixed relative to the cameras, the wire gripper-tiplocation can be obtained as part of the calibration. Optionally, thegripper tip location can be computed in a process before a wire isgripped by the wire gripper.

Once the image coordinates of the wire gripper and contact tips areknown in at least two camera views, the three-dimensional coordinates ofthe tips may be computed. To compute the three-dimensional coordinate ofa point, virtual lines may be formed that extend from a camera locationthrough the point in the image plane. These lines may be computed basedon the extrinsic parameters of the cameras, as obtained duringcalibration. The three-dimensional coordinate may be obtained as theleast-square solution that is closest to the virtual lines for at leasttwo camera views. The direction of a contact in the three-dimensionalend-effector coordinate frame may be computed as the vector differencebetween the contact tip three-dimensional location and the wire grippertip three-dimensional location.

The contact hole detection is imperative to properly identify thecorrect hole of the connector into which the wire contact is to beinserted. In each camera image including the connector, contact holesare detected. FIG. 9 illustrates the corresponding process flow whichmay be performed by the processing circuitry 36 of computing device 34.At the beginning of the process, the robot end-effector is positioned infront of a connector such that the connector surface is fully visible inat least two camera images from different perspectives. For at least twoof the camera images from different perspectives, the below-describedprocess is followed.

An image is acquired by a camera mounted to the robot end-effector at178. The image may be color filtered, such as using a red color filterby computing the R−(G+B)/2 for each pixel, where R, G, and B are thevalues for the red, green, and blue light channels respectively. Thecolor filter may be selected according to the color of a connector suchthat the filter best identifies differences in the connector surfacethat may correlate to holes of the connector. To crop the image, themedian image coordinate of the color-filtered intensity image iscomputed, and a window is cut out centered on the location of themedian, as shown at 180. Alternatively, the window may be centered atthe tip of the wire contact. The size of the window depends upon thetype of connector and may be a pre-specified parameter, such as a sizeof 270×270 pixels, sufficient to clearly identify each hole of theconnector. The color of the filter used should correspond to the colorof the connector.

The processing circuitry 36 may then be used to compute the squaredistance between a hole template and a local image patch from theintensity image for each patch location over the image as shown at 182.An example of a hole template may include an 18×18 pixel wide intensitygradient that mimics the shading inside a hole, where the intensityalong the gradient may follow a function f(x)=1/(1+exp(−x/1.7)). Theintensity of the template may be scaled to match minimum and maximumvalues of the color-filtered intensity image. This scaling increases therobustness to changes in lighting. The result of this operation mayinclude an intensity image in which low intensity areas (black) areareas of short distance to the hole template.

Using the intensity image may be used, such as by the processingcircuitry 36 of computing device 34, to isolate extrema identified inthe image. These extrema correspond to contact holes and are localminima in the distance image. To identify a local minimum, an ellipticalboundary around each pixel may be analyzed. The size of the boundary maydepend on the distance between neighboring holes on a connector. As anon-limiting example, the elliptical boundary may include half-axislengths of 25 and 15 pixels, where the longer axis is in the horizontaldirection approximating the elliptical shape of the holes in the cameraimage as the image is not coaxial with the connector. Pixels may bediscarded as extrema candidates if they have one or more pixels in theirboundary with an intensity below the candidate pixel's intensity times aconstant factor greater than one. This constant factor may depend on theconnector type. For example, the constant factor may be around 1.7. Afactor larger than one ensures that the extrema are more pronounced andmay eliminate or reduce false detections. For each remaining candidatepixel, a weighted average may be computed over all pixels inside itselliptical boundary, where the weight is the inverse of the intensity ofeach pixel in the distance image.

If a total weight computed over all pixels inside an elliptical boundaryis above a threshold (e.g., 2), then the weighted average may beidentified as a hole and added to the list. The detection of isolatedpeaks is shown at 184 whereby holes of the connector are identified. Toavoid that pixels of the same hole are counted as separate holes (doublecounting), all pixels inside an ellipse used for weighted averaging maybe marked and automatically discarded as extrema candidates. The resultof this operation is a list of contact holes. As an additionaloperation, outliers may be removed from the hole list. To removeoutliers, first the minimum distance (d_(min)) may be computed betweentwo holes. Second, any hole may be discarded as an outlier that has adistance to its nearest neighbor hole that is larger than a constantfactor times d_(min) (e.g., the constant factor of 2).

The hole list may be matched against known hole locations from technicalspecifications and/or drawings of the connector. This matching cancompensate for missed holes and allow for assignment of holeidentification numbers to the detected holes. From the technical drawingof a connector, a two-dimensional mask of hole locations may beextracted. This mask may include a list of contacts with the identitiesand locations in a connector-centered coordinate system. To match themask to the hole list, the mask may be rotated (in three axes) andtranslated (in three directions) in the end-effector coordinate systemsuch that it optimally overlaps with the hole list. To compute theoverlap, the mask may be projected onto each camera image using theparameters from the camera calibration. The cost function foroptimization may be the sum of square distance between the holes fromthe list and their closest neighboring projected holes. A non-limitingexample of an algorithm to optimize this cost function may includePowell's method.

The aforementioned processes provide, for each camera image analyzed,the line describing the wire contact and the location of the targethole. Based on this information, the corrective movement for the robotend-effector can be computed. The target hole location in two or morecamera images is identified at 200. The three-dimensional location “p”of the target hole in the end-effector coordinate system is computed at202. To compute this location, an optimization algorithm is used thatminimizes the sum of square distances between the target holetwo-dimensional image locations and the projections of thethree-dimensional location on to the camera images. A non-limitingexample for an optimization includes Powell's method. Here, thethree-dimensional location may be constrained to lie in the plane of theconnector surface. This plane may be known due to the mask-optimizationprocess described above, which rotates and translates the mask to matchthe connector surface.

A location “r” is computed in the end-effector coordinate system thatprojects closest to the contact line in each image as shown at 204. Thislocation may be also constrained to lie in the plane of the connectorsurface. An optimization algorithm may be used to compute “r”. Based onthe resulting values of “p” and “r”, the corrective movement may becomputed at 208 as c=p−r. The movement of the end-effector may becarried out at 210.

The identified target hole of the connector for the wire contact maythen be utilized to facilitate insertion of wire ends into respectivewire contact insertion holes of the connector. In this regard, a wiremay be identified by a wiring diagram or the like to be inserted into aparticular wire contact insertion hole of the connector (and, in someembodiments, also into electrical contact with a respective wire contactthat is aligned with the wire contact insertion hole) with theparticular wire contact insertion hole being identified by a contact IDnumber, which may be identified on the connector via the aforementionedmap of identifiers for the connector. Prior to insertion into the wirecontact insertion hole of the connector, a wire contact is generallyconnected to, e.g., crimped upon, a bare end of the wire to form a wireend. Based upon the contact ID numbers and corresponding locations ofthe candidate contact insertion hole for the connector 10, a wire endmay be inserted into the connector at the location associated with acontact insertion hole having the contact ID number of the wire contactinsertion hole 18 into which the wire is to be inserted. The computingdevice 34, such as the processing circuitry 36, may be configured todetermine the candidate contact insertion hole into which a wire is tobe inserted based upon the contact ID number of a candidate contactinsertion hole, such as based upon correspondence between the contact IDnumber of a candidate contact insertion hole and the contact ID numberof the wire contact insertion hole 18 into which the wire end is to beinserted as defined by a wiring diagram or the like. The computingdevice 34, such as the processing circuitry 36, is also configured todetermine the position of a robot 44 and, more particularly, a roboticend-effector utilized to insert the wire end into the candidate contactinsertion hole based upon the location of the candidate contactinsertion hole in the connector-based coordinate system and using thealignment methods described above for efficient and repeatable insertionof wires into corresponding holes of the connector.

As such, the computing device 34, such as the processing circuitry 36,may effectively drive a robot 44, such as a robotic end-effector, orotherwise provide information, such as a list of candidate contactinsertion holes, contact ID numbers and corresponding locations in theconnector-based coordinate system, to the robot sufficient to drive therobotic end-effector in such a manner as to insert the wire ends of awire bundle assembly into corresponding wire contact insertion holes 18.See, for example, FIG. 11 in which a plurality of wires 90 have beeninserted into respective wire contact insertion holes 18 of theconnector 10 in order to establish mechanical connection between thewire ends and the connector 10. By facilitating the automation of theconnection process associated with a wire bundle assembly, the system30, method and computer program product of an example embodimentincrease the efficiency with which wire ends of a wire bundle assemblymay be mechanically connected to a connector 10 and correspondinglyreduce the error rate and cost of the resulting assembly.

While the aforementioned process involves aligning a single wire contactfor insertion, embodiments of the present disclosure may be used toalign and insert a plurality of wire contacts into a respectiveplurality of target holes of a connector. However, the order in whichwire contacts are inserted into target holes may be established in sucha manner as to not diminish the effectiveness of the alignment methodsdescribed herein. As discussed above, the wire contact and the targethole must each be visible in at least two camera images from twodifferent perspectives for proper alignment. If wire contacts areinserted into the connector in an improper order, a target hole of theconnector may be obstructed from view of one or more cameras. As such,an order of assembly may include starting with target holes of theconnector which are furthest from the cameras, such as the bottom of theconnector in the example configuration shown herein. In this manner,wires will be inserted to the connector from the bottom-up to avoid aninserted wire obstructing the camera view of a target hole. A pluralityof cameras from a plurality of different perspectives may mitigate theinstallation order requirement as when a camera view of a target hole isobstructed, provided the target hole remains visible in at least twoimages from at least two perspectives, the process described herein canbe performed effectively.

FIG. 12 is a flowchart of a process for aligning a wire contact with atarget hole of a connector according to an example embodiment of thepresent disclosure. As shown, images are obtained from at least twoimage acquisition devices, such as cameras 32 of system 30, attached toan end-effector of a robot, where the images are of a wire gripper ofthe end effector, at 220. At 222, a wire contact is detected, such as byprocessing circuitry 36 of computing device 34, within at least oneimage from each of the at least two image acquisition devices. Within atleast one image from each of the at least two image acquisition devices,one or more insertion holes of the connector are detected at 224, suchas by processing circuitry 36 of computing device 34. Correctivemovement of the robot end effector is identified at 226, such as byprocessing circuitry 36 of computing device 34, that aligns a targethole of the one or more insertion holes of the connector with the wirecontact. At 228, the robot (44 of system 30 of FIG. 3) is caused, suchas by processing circuitry 36 via communications interface 42, to movethe end-effector according to the identified corrective movement.

Once the alignment of the wire contact with the target hole of theconnector is performed, the wire contact may be inserted into the targethole for assembly of the connector and wire bundle thereof. Initially, awire contact may be moved by the wire gripper of the end-effector of therobot to a preparation position proximate the target hole. The alignmentis performed in a plane parallel to the surface of the connector, whileinsertion is performed on an axis orthogonal to the plane of the surfaceof the connector.

During insertion, the robot end effector 100 and/or the wire gripper 108may include one or more sensors for determining one or more forcesacting on the wire 111 or the wire contact 114. Forces may be sensed byvirtue of resistance encountered by the motive force of the robot (e.g.,a servo motor, a hydraulic pump, etc.). Forces may optionally be sensedby a strain gauge arrangement which may be disposed on the wire gripperand configured to sense resistance to movement of the wire gripper orwire/wire contact held therein. Various other force sensing arrangementsmay be employed as necessary to determine forces acting on the wirecontact 114 of the wire 111 held by the wire gripper 108.

FIG. 13 illustrates a process flow according to the aforementioneddescribed method of alignment and insertion of a wire contact with atarget hole of a connector. As shown, the process begins at 300 with aconnector mounted and ready to receive a wire contact in a target hole.The wire including the wire contact is loaded at 302 into a wire gripper(e.g., 108 of FIG. 4) of an end-effector (e.g. 100) of a robot. Initialvision processing is performed at 304, such as using cameras 102 and 104of FIG. 4. The end-effector may then move the wire gripper and wire tothe preparation position 306. At 308, alignment of the wire contact withthe target hole is performed, such as by using the process detailedabove. As shown, the corrective transformation is computed at 310, usingthe process described in detail above. If the correction transformationis below a threshold amount, the alignment is considered complete andthe process continues. If the correction transformation is above athreshold, the number of corrective transformations already completed ischecked at 312, and if it is below a threshold number, the correctivetransformation occurs at 314. If the number of correctivetransformations exceeds a predefined number, a failure may be identifiedat 330. However, if the corrective transformation is successful at 314and results in an alignment below a threshold corrective transformation,then the wire gripper holding the wire contact is moved toward theconnector at 316.

The process of FIG. 13 continues at 318 whereby movement of the wiregripper toward the connector to a position in which misalignment wouldcause contact but not damage to the wire contact or connector. If forcefeedback on the wire gripper exceeds a threshold value, it may bedetermined that the wire contact is not aligned with the target hole,such that the wire contact is retracted at 320 and alignment is againperformed at 308. If the force feedback at the wire gripper is below athreshold, the wire contact is established as being aligned with thetarget hole. In this way, the wire gripper is prevented from damagingthe connector during insertion of the wire contact.

If the wire contact is established to be at the target hole of theconnector whereby operation 318 is established to be true, the processof insertion continues as shown in FIG. 14. As shown, the insertionprocess begins at 322. During insertion, the robot end effector and/orwire gripper monitors the insertion force, F, at 324. If the force isabove a predetermined value at 326, for example 16 Newtons, a valuewhich may be dependent on the type of contact and the type of connector,the insertion may be temporarily halted. Once temporarily halted, adetermination is made with respect to the depth of the insertion at 328.The insertion depth can is estimated based on the initial distance tothe connector and the travel distance of the robot end effector. Theinitial distance can be estimated through vision, such as through theimage processing described above identifying the location of theconnector relative to the wire connector. If the depth of insertion d isabove a minimum depth d_(min), a pull test is conducted at 332. The pulltest will be described further below.

If the depth of insertion d is less than a minimum depth d_(min), thisis an indication that the contact is stuck and an error correctionmaneuver is needed at 352. The error correction maneuver is describedfurther below. The error correction maneuver is carried out only if thenumber of such error corrections, c, is below a threshold number,c_(min), which may be, for example, five attempts. This determination isshown at 334. If the number of correction attempts is greater than thethreshold number c_(min), the robot end effector and wire gripper movethe wire contact outside of the connector hole 336 and the alignment ofthe wire contact with the hole is repeated at 338. This may be thealignment process of 308 of FIG. 13, for example. After the contactdirection alignment has been corrected at 338, the wire connector ismoved to the surface of the connector at 340 and the number of insertionattempts i, is incremented at 342 before insertion begins again at 322.This process is repeated until the pull test 332 is executed, or themaximum number of insertion attempts (i_(max)) has been reached at 344.When starting the re-alignment process of 338, the number of correctionsc is reset to zero. If the pull test 332 passes at 348, the insertion iscompleted with a successful insertion 350. If the pull test fails, thetool head moves the contact again outside of the connector 336 forrealignment 338 unless the maximum number of insertion attempts i hasbeen reached. Once the number of insertion attempts reaches a maximumnumber (i_(max)), the wire contact insertion process is deemed to havefailed and the process is aborted at 360.

The pull test operation may be performed to confirm seating of the wirecontact within the connector as shown above at operation 332. For thepull test, the wire gripper may pull back on the wire, away from theconnector, until a specific distance or force threshold is reached. Ifthe force threshold is reached before the specified distance, then thewire contact is confirmed as properly seated. If the specified distanceis reached before the force threshold is achieved, the wire insertionfailed as the wire is determined to not be fully seated.

The error correction maneuver of operation 352 provides an automatedcorrection of the wire contact direction. FIG. 15 illustrates aflowchart of the process for the error correction maneuver. When theinsertion force is above the predefined threshold F. of operation 326 inFIG. 14, the insertion is stopped at 364. With the depth less than aminimum insertion depth of operation 328 and the number of errorcorrection maneuvers c is below the maximum of c_(max) attempts, theerror correction maneuver begins with retraction of the wire contactslightly out of the connector at 366 but still partially inserted withinthe connector. An orientation of the wire contact is determined at 368.The orientation may be determined based on images acquired fromdifferent angles of the wire contact and the connector, such has withcameras 102 and 104 capturing images of the wire contact 114 and theconnector 110. Based on the wire contact direction established in theimages, the orientation of the wire contact may be projected onto theplane perpendicular to the orientation of the wire gripper at 370. Thisprojection of the orientation is a contact direction vector inthree-dimensional Cartesian space. The contact direction vector lies onthe two-dimensional plane perpendicular to the wire gripper and at leastapproximately parallel to the connector surface. This projection isperformed such that the corrective movement is made in the plane of theconnector surface. The projected vector (V) is multiplied by amultiplier that is greater than one (e.g., 1.5), which results in acorrective movement step (V*1.5). This corrective movement step isincreased as shown at 372 to improve the probability of moving thecontact around any obstacles internal to the connector. The wire gripperis moved as defined by the resulting vector (1.5*V) at 374, and theinsertion is resumed at 376 corresponding to insertion 322 of FIG. 14.

The determination of the orientation of the wire contact at operation368 is important to the success of the error correction maneuver. Themain operations using two or more camera images are illustrated in FIG.16, while the flowchart of FIG. 17 illustrates elements of the processof determining the orientation of operation 368 in FIG. 15. As shown inFIG. 16, two or more images from at least two different angles are usedas input at 378. The image region near the wire gripper tip is croppedat 380. The size of this region may be, for example, 200 by 200 pixels,while the camera field of view may be 1600 by 1200 pixels. Gold-huepixels are found in the images at 382. While gold-hue pixels are foundin the example embodiment, other hues may be used based on thecomponents of the wire contact, such as silver. Pixels of the chosencolor can be extracted by first converting the cropped images to HSV(Hue, Saturation, Value) color space, and subsequently thresholding theHSV values. For example: 20<=H<=28; 100<=S<=180; and 80<=V<=255, wherethe maximum range of H is - to 180, S is - to 255, and V is - to 255. Toremove noise, erosion and dilation operations may be carried out. Theprocess flow is shown in greater detail in FIG. 17, where the pixels ofthe image are eroded at 402, and pixels remaining are dilated at 404.The parameter for these operations may be a kernel size of 4-by-4pixels, for example.

A line is fitted to the resulting pixels at 406 in the process flow ofFIG. 17. To make the line fit more robust to single outlier pixels, aloss function may be employed that increases less than quadratic withthe distance from the line for pixels that are beyond a certainthreshold distance. For example, the Huber distance can be used asdefined by:

${\rho(r)} = \{ \begin{matrix}{{r^{2}/2}\ } & {{{if}\mspace{9mu} r} < C} \\{{C \cdot ( {r - {C/2}} )}\ } & {otherwise}\end{matrix} $

Where C=30 pixels, and r is the distance to the fitted line.

Edge detection is then performed at 384 in the cropped images. The edgedetection may be limited to pixels that are within a certain distance(e.g., 15 pixels) of the above-described fitted line. This limitationprevents the image detection from picking up edges from wires that mightbe located below the contact (e.g., with previously inserted wirecontacts). The edges are further cleaned up through edge removal ofedges that are perpendicular to the fitted line at 408 of the processflow of FIG. 17. An example algorithm may include Canny Edge Detection,for example.

A Hough transform may be carried out on the edges of the image as shownat 410 of the process flow of FIG. 17. The Hough transform extractslines in the image. From all of the detected lines, the dominant line ischosen (e.g., the longest line) at 386 of both FIGS. 16 and 17. Thisdominant line may be averaged with neighboring lines that are in thesame direction and have a length above a threshold (e.g., 50% of themaximum value from the Hough transform). The resulting line is anestimate of the two-dimensional direction of a contact in the cameraimage. The two-dimensional directions from the at least two images areused to compute the three-dimensional vector at 388 as an estimate ofthe contact direction. To compute the three-dimensional vector, thecamera(s) capturing the images may require calibration (e.g., theintrinsic and extrinsic camera parameters have to be known or estimatedbefore the insertion). To estimate the three-dimensional vector, twopoints are chosen on each two-dimensional line. Virtual rays are formedfrom the camera location through these points in the image plane. Foreach line, two rays span a plane in three dimensions. The intersectionof the two planes (from two cameras) in three-dimensions is the desiredestimated three-dimensional direction of the contact in the Cartesianspace of the robot end effector.

The process flow of FIG. 17 further illustrates another example of amethod of identifying the wire contacts in the two or more images. Asshown at 378, the images are acquired and cropped at 380. In anembodiment in which color images of the wire contact and connector areacquired, the image associated with each different color channel of thecameras, such as the red and green color channels may be averaged tocreate a composite image for subsequent analysis and review. The averageof the red and green components of the image are computed at 404, andthe image is blurred at 406, and processed in grayscale at 408 ahead ofedge detection at 384. This process may be used for various connectortypes not requiring a gold, silver, or specific color of pixel to beextracted.

FIG. 18 illustrates an example method for correction of automatedinsertion of a wire contact into a target hole of a connector. As shownat 400, a robot having an end-effector is controlled to align the wirecontact with the target hole of the connector. The robot is thencontrolled to advance the wire contact toward the target hole of theconnector and at least partially insert the wire contact into the targethole at 410. The robot ceases insertion at 420 in response to a forcebetween the wire contact and the connector exceeding a predeterminedvalue. A depth of insertion of the wire contact within the target holeis determined at 430, such as by using visual verification through imageacquisition devices. A determination is then made at 440 as to whetherthe depth of insertion is above a minimum depth. If so, the wire contactis considered properly inserted and seated, and a pull test of the wirecontact is performed at 450. If the depth is below the minimum, an errorcondition is identified at 460 using visual feedback. A determination isthen made at 470 as to whether too many corrective actions have alreadybeen performed on the wire contact. If the number of corrective actionsis below a predefined number, an error correction operation is performedat 480. If too many corrective actions have already been performed, thewire contact is moved away from the target hole at 490 for potentialmanual intervention, a re-setting or re-calibration of the system, orother operation to be performed.

As described above, FIGS. 6, 7, 10, 12-15, 17, and 18 illustrateflowcharts of a system 30, method, and computer program productaccording to example embodiments of the present disclosure. It will beunderstood that each block of the flowcharts, and combinations of blocksin the flowcharts, may be implemented by various means, such ashardware, firmware, processor, circuitry, and/or other devicesassociated with execution of software including one or more computerprogram instructions. For example, one or more of the proceduresdescribed above may be embodied by computer program instructions. Inthis regard, the computer program instructions which embody theprocedures described above may be stored by the memory 38 of a system 30employing an embodiment of the present disclosure and executed by theprocessing circuitry 36 of the system 30. As will be appreciated, anysuch computer program instructions may be loaded onto a computer orother programmable apparatus (e.g., hardware) to produce a machine, suchthat the resulting computer or other programmable apparatus implementsthe functions specified in the flowchart blocks. These computer programinstructions may also be stored in a computer-readable memory that maydirect a computer or other programmable apparatus to function in aparticular manner, such that the instructions stored in thecomputer-readable memory produce an article of manufacture the executionof which implements the function specified in the flowchart blocks. Thecomputer program instructions may also be loaded onto a computer orother programmable apparatus to cause a series of operations to beperformed on the computer or other programmable apparatus to produce acomputer-implemented process such that the instructions which execute onthe computer or other programmable apparatus provide operations forimplementing the functions specified in the flowchart blocks.

Accordingly, blocks of the flowcharts support combinations of means forperforming the specified functions and combinations of operations forperforming the specified functions for performing the specifiedfunctions. It will also be understood that one or more blocks of theflowcharts, and combinations of blocks in the flowcharts, can beimplemented by special purpose hardware-based computer systems whichperform the specified functions, or combinations of special purposehardware and computer instructions.

In some embodiments, certain ones of the operations above may bemodified or further amplified. Furthermore, in some embodiments,additional optional operations may be included. Modifications,additions, or amplifications to the operations above may be performed inany order and in any combination.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art having the benefit ofthe teachings presented in the foregoing descriptions and the associateddrawings. Therefore, it is to be understood that the present applicationis not to be limited to the specific embodiments disclosed and thatmodifications and other embodiments are intended to be included withinthe scope of the appended claims. Moreover, although the foregoingdescriptions and the associated drawings describe example embodiments inthe context of certain example combinations of elements and/orfunctions, it should be appreciated that different combinations ofelements and/or functions may be provided by alternative embodimentswithout departing from the scope of the appended claims. In this regard,for example, different combinations of elements and/or functions thanthose explicitly described above are also contemplated as may be setforth in some of the appended claims. Although specific terms areemployed herein, they are used in a generic and descriptive sense onlyand not for purposes of limitation.

That which is claimed is:
 1. A system for correction of automatedinsertion of a wire contact into a target hole of a connector, thesystem comprising: a robot having an end-effector, wherein theend-effector comprises a wire gripper; a computing device, wherein thecomputing device is configured to: control the robot to align the wirecontact with the target hole of the connector; control the robot toadvance the wire contact toward the target hole of the connector and atleast partially insert the wire contact into the target hole; controlthe robot to cease insertion in response to a force between the wirecontact and the connector exceeding a predefined value; determine adepth of insertion of the wire contact within the target hole of theconnector; in response to the depth of insertion being above apredetermined depth, control the robot to perform a pull test of pullingthe wire contact away from the connector; in response to the depth ofinsertion being below a predetermined depth: identify an error conditionusing visual feedback; determine a number of corrective operationsperformed; perform an error correction operation in response to thenumber of corrective operations being below a predefined number; andmove the wire contact away from the target hole in response to thenumber of corrective operations being above the predefined number. 2.The system of claim 1, wherein the computing device configured toperform an error correction operation comprises the computing deviceconfigured to: control the robot to adjust an orientation of the wirecontact to be oriented perpendicular to a front surface of theconnector; and control the robot to re-insert the wire contact into thetarget hole of the connector.
 3. The system of claim 2, wherein thecomputing device configured to adjust the orientation of the wirecontact to be oriented perpendicular to the front surface of theconnector comprises the computing device configured to: determine theorientation of the wire contact relative to the connector; project theorientation of the wire contact onto a plane perpendicular to anorientation of the wire gripper to identify a resulting vector; andcontrol the robot to adjust the orientation of the wire contactaccording to the resulting vector to be oriented perpendicular to thefront surface of the connector.
 4. The system of claim 2, wherein thecomputing device configured to adjust the orientation of the wirecontact to be oriented perpendicular to the front surface of theconnector comprises the computing device configured to: determine theorientation of the wire contact relative to the connector; project theorientation of the wire contact onto a plane perpendicular to anorientation of the wire gripper to identify a resulting vector; increasethe resulting vector by a predetermined amount; and control the robot toadjust the orientation of the wire contact according to the increasedresulting vector to be oriented perpendicular to the front surface ofthe connector.
 5. The system of claim 1, wherein the computing deviceconfigured to move the wire contact away from the target hole furthercomprises the computing device configured to: control the robot tore-align the wire contact with the target hole; control the robot toadvance the wire contact toward the target hole of the connector and atleast partially insert the wire contact into the target hole; controlthe robot to cease insertion in response to a force between the wirecontact and the connector exceeding a predefined value; determine adepth of insertion of the wire contact within the target hole of theconnector; in response to the depth of insertion being above apredetermined depth, control the robot to perform a pull test of pullingthe wire contact away from the connector; and provide for indication ofsuccessful insertion of the wire contact into the target hole of theconnector in response to the pull test satisfying predeterminedcriteria.
 6. The system of claim 5, wherein the predetermined criteriacomprises movement of the wire contact less than a predefined amount outof the target hole in response to a pull force applied to the wirecontact of at least a predefined force.
 7. The system of claim 1,wherein the end effector further comprises one or more image acquisitiondevices for capturing images of the wire contact and the connector fromat least two different angles.
 8. The system of claim 7, wherein thecomputing device configured to control the robot to align the wirecontact with the target hole of the connector comprises the computingdevice configured to: acquire images from at least two different anglesusing the one or more image acquisition devices; identify a location ofthe target hole; identify an alignment direction to align the wirecontact with the target hole of the connector; and control the robot tomove the wire contact in the alignment direction in a plane orthogonalto an axis along which the wire contact extends.
 9. The system of claim1, wherein the depth of insertion of the wire contact into the targethole of the connector is determined based on an initial distance of thewire contact to a surface of the connector and a traveled distance ofthe robot.
 10. The system of claim 9, wherein the initial distance ofthe wire contact to the surface of the connector is established usingimages of the wire contact and the connector from at least two differentangles.
 11. A method for correction of automated insertion of a wirecontact into a target hole of a connector, the method comprising:controlling a robot having an end-effector to align the wire contactwith the target hole of the connector using a wire gripper of theend-effector; controlling the robot to advance the wire contact towardthe target hole of the connector and at least partially insert the wirecontact into the target hole; controlling the robot to cease insertionin response to a force between the wire contact and the connectorexceeding a predefined value; determining a depth of insertion of thewire contact within the target hole of the connector; in response to thedepth of insertion being above a predetermined depth, controlling therobot to perform a pull test of pulling the wire contact away from theconnector; in response to the depth of insertion being below apredetermined depth: identifying an error condition using visualfeedback; determining a number of corrective operations performed;performing an error correction operation in response to the number ofcorrective operations being below a predefined number; and moving thewire contact away from the target hole in response to the number ofcorrective operations being above the predefined number.
 12. The methodof claim 11, wherein performing an error correction operation comprises:controlling the robot to adjust an orientation of the wire contact to beoriented perpendicular to a front surface of the connector; andcontrolling the robot to re-insert the wire contact into the target holeof the connector.
 13. The method of claim 12, wherein adjusting theorientation of the wire contact to be oriented perpendicular to thefront surface of the connector comprises: determining the orientation ofthe wire contact relative to the connector; projecting the orientationof the wire contact onto a plane perpendicular to an orientation of thewire gripper to identify a resulting vector; and controlling the robotto adjust the orientation of the wire contact according to the resultingvector to be oriented perpendicular to the front surface of theconnector.
 14. The method of claim 12, wherein adjusting the orientationof the wire contact to be oriented perpendicular to the front surface ofthe connector comprises: determining the orientation of the wire contactrelative to the connector; projecting the orientation of the wirecontact onto a plane perpendicular to an orientation of the wire gripperto identify a resulting vector; increasing the resulting vector by apredetermined amount; and controlling the robot to adjust theorientation of the wire contact according to the increased resultingvector to be oriented perpendicular to the front surface of theconnector.
 15. The method of claim 11, wherein moving the wire contactaway from the target hole further comprises: controlling the robot tore-align the wire contact with the target hole; controlling the robot toadvance the wire contact toward the target hole of the connector and atleast partially insert the wire contact into the target hole;controlling the robot to cease insertion in response to a force betweenthe wire contact and the connector exceeding a predefined value;determining a depth of insertion of the wire contact within the targethole of the connector; in response to the depth of insertion being abovea predetermined depth, controlling the robot to perform a pull test ofpulling the wire contact away from the connector; and providing forindication of successful insertion of the wire contact into the targethole of the connector in response to the pull test satisfyingpredetermined criteria.
 16. An apparatus comprising at least oneprocessor and at least one memory including computer program code, theat least one memory and computer program code configured to, with theprocessor, cause the apparatus to at least: control a robot having anend-effector to align a wire contact with a target hole of a connectorusing a wire gripper of the end-effector; control the robot to advancethe wire contact toward the target hole of the connector and at leastpartially insert the wire contact into the target hole; control therobot to cease insertion in response to a force between the wire contactand the connector exceeding a predefined value; determine a depth ofinsertion of the wire contact within the target hole of the connector;in response to the depth of insertion being above a predetermined depth,control the robot to perform a pull test of pulling the wire contactaway from the connector; in response to the depth of insertion beingbelow a predetermined depth: identify an error condition using visualfeedback; determine a number of corrective operations performed; performan error correction operation in response to the number of correctiveoperations being below a predefined number; and move the wire contactaway from the target hole in response to the number of correctiveoperations being above the predefined number.
 17. The apparatus of claim16, wherein causing the apparatus to perform an error correctionoperation comprises causing the apparatus to: control the robot toadjust an orientation of the wire contact to be oriented perpendicularto a front surface of the connector; and control the robot to re-insertthe wire contact into the target hole of the connector.
 18. Theapparatus of claim 17, wherein causing the apparatus to adjust theorientation of the wire contact to be oriented perpendicular to thefront surface of the connector comprises causing the apparatus to:determine the orientation of the wire contact relative to the connector;project the orientation of the wire contact onto a plane perpendicularto an orientation of the wire gripper to identify a resulting vector;and control the robot to adjust the orientation of the wire contactaccording to the resulting vector to be oriented perpendicular to thefront surface of the connector.
 19. The apparatus of claim 16, whereincausing the apparatus to control the robot to align the wire contactwith the target hole of the connector comprises causing the apparatusto: acquire images from at least two different angles using one or moreimage acquisition devices; identify a location of the target hole;identify an alignment direction to align the wire contact with thetarget hole of the connector; and control the robot to move the wirecontact in the alignment direction in a plane orthogonal to an axisalong which the wire contact extends.
 20. The apparatus of claim 16,wherein causing the apparatus to move the wire contact away from thetarget hole further comprises causing the apparatus to: control therobot to re-align the wire contact with the target hole; control therobot to advance the wire contact toward the target hole of theconnector and at least partially insert the wire contact into the targethole; control the robot to cease insertion in response to a forcebetween the wire contact and the connector exceeding a predefined value;determine a depth of insertion of the wire contact within the targethole of the connector; in response to the depth of insertion being abovea predetermined depth, control the robot to perform a pull test ofpulling the wire contact away from the connector; and provide forindication of successful insertion of the wire contact into the targethole of the connector in response to the pull test satisfyingpredetermined criteria.